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(57) Abstract 

An improved visual user interface enables control of the interaction of a device with a spatial region. The visual user interface 
includes a display of a control space and a display of a target space. The content of the displayed target space can be a representation of 
some or all of the spatial region. The content of the displayed control space and displayed target space can also be esttablished such that the 
displayed target space provides context for the displayed control space. The device is operated in accordance with the state of the displayed 
control space. The visual user interface enables a user to change the display of the control space, thereby controlling the operation of the 
device. Such a visual user interface in which a target scene provides context for a control scene can, for example, facilitate perception by 
a video camera operator of a current filmed scene in the context of the current filmed scene's surroundings, thereby enhancing the camera 
operator's ability to engage in critical and sophisticated camera work with great precision. The visual user interface can also include a 
space for making annotations regarding the interaction of the device with the spatial region, thereby enabling such annotations to easily be 
made contemporaneously with control of the device, so that a single person can perform bodi functions well. Additionally, the annotation 
capability can be implemented in a way that enables the user to hand write the annotations, thus facilitating a wide range of expression in 
creating annotations and avoiding the problems associated with voice annotation systems. The invention can be used in a wide variety of 
applications, such as, for example, applications which require the control of a video device. 
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VISUAL USER INTERFACE FOR CONTROLLING THE INTERACTION OF A DEVICE WITH A SPATIAL REGION 

BACKGROUND OF THE INVENTION 
5 1. Field of the Invention 

This invention relates to a visual user interface" for 
use in controlling the interaction of a device with a spatial 
region, such as the acquisition or display by a video device 
of a sequence of filmed scenes that are each part of a 

10 panoramic scene, 
2 . Related Art 

A conventional video camera typically includes a 
.viewfinder, the view through which corresponds generally to a 
portion (the filmed scene) of a scene that is being filmed, 

15 During conventional operation of such a video camera, the 

camera operator looks through the viewfinder while filming so 
that camera operator knows what is being filmed and so that 
the camera operator can control the camera to film a desired 
sequence of filmed scenes. However, during such operation, 

20 the camera operator's view is limited to what can be seen 
through the viewfinder. Since the camera operator cannot 
view portions of a scene adjacent to the filmed scene, the 
camera operator is deprived of context for the filmed scene. 
This limitation impairs the camera operator's ability to 

25 operate the camera so as to film a scene in a desired way 
(i.e. to acquire a desired sequence of filmed scenes) . It 
has historically been part of the art of a skilled camera 
operator to be able to remember the content of a scene having 
a scope larger than can be encompassed within the field of 

3 0 view of the viewfinder of the video camera, so that the 

camera operator can seamlessly control the camera to change 
the filmed scene in a desired manner, 

A video camera can be hand-held. A video camera can 
also be mounted on a camera support device that typically 
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provides multiple degrees of freedom of movement of the 
camera. The above limitation in operation of a conventional 
video camera applies to both hand-held and mounted camera 
operation in which the camera operator controls the content 
5 of the filmed scene by looking through the viewfinder. While 
the camera operator can periodically look away from the 
viewfinder to observe the larger context d"f the scene being - 
filmed, such operation is undesirable for a variety of 
reasons. For example, the camera operator may inadvertently 

10 misdirect the camera while not looking through the 

viewfinder. It is also difficult for the camera operator to 
readjust image perception and focus each time the operator 
looks away from the viewfinder to the larger scene, and vice 
versa. Additionally, the inability to view a current filmed 

15 scene simultaneously with the surrounding context of the 
current filmed scene makes it difficult for the camera 
operator to plan movement of the camera to film other scenes 
adjacent to the current filmed scene. 



20 display at a remote site is connected to the video camera so 
that a current filmed scene is displayed by the remote video 
display. A control device is connected to, for exajuple , one 
or more motors (or other driving devices) which are, in turn, 
coupled to the camera support so as to move the camera 

25 support in response to signals received from the control 
device. The camera operator observes the remote video 
display and operates the control device to control the 
sequence of filmed scenes acquired by the video camera. 
Aside from the obvious benefits (e.g., the capability of 

30 filming in areas that are hazardous to humans) , remote video 
camera operation allows the camera operator to move away from 
the viewfinder so that the entire area surrounding a current 
filmed scene can be more easily observed during filming. 
However, the camera operator still cannot simultaneously view 

3 5 the surrounding area and the current filmed scene, so that 



Mounted cameras can also be operated remotely. 



A video 
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such remote camera operation still suffers from the above- 
described deficiencies to an undesirable degree. 

The above -described operation of a video camera is 
"manual" operation, i.e., the camera operator must either 
5 directly or indirectly provide control of the camera at each 
instant in time. Automatic control of a video camera has 
also previously been implemented. At a predetermined 
location, a video camera is mounted on a camera support that 
allows a variety of camera motions. The video camera is 
10 connected to a control device as described above. Metrics 
are established for the camera support so that positions of 
the camera support can be correlated with particular filming 
directions of the video camera. During operation of the 
video camera, the control device, in accordance with a pre- 
15 specified instructions, transmits control signals to the 

motors to cause the camera support to be moved in a fashion 
that produces a desired sequence of filmed scenes. _ Using 
such automatic control, the position, velocity and 
acceleration of the video camera can be controlled at each 
20 instant in time. Automatic control of a video camera can 
enable dynamic control of the camera that may not be 
reproducible by a human operator with the same precision or 
speed. Automatic control can also enable pre-scripting of a 
sequence of filmed scenes. However, automatic control still 
25 does not satisfactorily address the above-described problems, 
since it does not allow real-time camera control in a manner 
that facilitates perception by the camera operator of the 
filmed scene in the context of the filmed scene's 
surroundings. Filming a live event, such as a sporting 
3 0 event, is just one example of a situation in which such 
automatic camera control is inappropriate. 

Once a videotape (or other set of video data) has been 
produced by filming a sequence of video scenes, it is 
desirable to have an indication of the content of the 
35 videotape and, further, an indication of where (i.e., when) 
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particular content occurs within the videotape, A convenient 
way to create a record of the content of a videotape is to 
produce a set of time-identified annotations during filming 
of the videotape. However, many videotaping systems do not 
5 include a capability to produce such annotations. Creation 
of a handwritten set of annotations by the camera operator 
using, for example, paper and pencil can unduly interfere 
with the actual filming, thus necessitating the presence of a 
second person to make the annotations. Some videotaping 

10 systems include a user interface that enables selection of 
one of a set of predetermined textual annotations. However, 
such systems can be inadequate because they necessarily 
enable only a limited range of description of the content of 
the videotape. Further, these systems, while typically more 

15 convenient than using paper and pencil, may also require the 
presence of a second person to perform the annotation so that 
the camera operator is not inordinately distracted from 
filming. Some videotaping systems enable voice annotation to 
be made during filming of a videotape. However, voice 

20 annotation is inappropriate for situations in which speaking 
during filming of the videotape may disrupt the event or 
scene being filmed. Further, searching a set of audio data 
(here, the set of spoken annotations) for desired content can 
be difficult, therefore making it difficult to review a set 

25 of annotations to find a particular description of the 
videotape content . 

SUMMARY OF THE INVENTION 

In accordance with one aspect of the invention, control 
of the interaction of a device with a spatial region is 
3 0 enabled using an improved visual user interface. The visual 
user interface includes a display of a control space and a 
display of a target space. The target space typically has a 
specified relationship to the spatial region and, often, is a 
representation of some or all of the spatial region. The 
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content of the control space and target space can also be 
(and often is) established such that the target space 
provides context for the control space. The device is 
operated (i.e., interacts with the spatial region) in 
5 accordance with the state of the display of the control 

space. For example, the device can be operated in accordance 
with the' spatial relationship "between the control space and - 
the target space. The visual user interface according to 
this aspect of the invention enables a user to change the 
10 state of the display of the control space, thereby 

controlling the operation of the device. Generally, the 
display of the control space can be changed in any manner 
that is possible for the application with which the visual 
user interface is being used. 
15 For example, a visual user interface according to this 

aspect of the invention can display a target scene and a 
control scene. Such a visual user interface can be used, for 
example, to control a video device. The video device 
controlled can be of any type, including video devices that 
20 acquire video data (e.g., video cameras, video-enabled motion 
picture cameras) and video devices that display video data 
(e.g., televisions, computer display monitors, and video or 
graphics projection systems). A user can, for example, cause 
the control scene to change in size, shape, angular 
25 orientation and/or location with respect to the target scene, 
which can cause, for example, a filmed scene to change in a 
corresponding way. Other aspects of the control scene, such 
as visual parameters (e.g., focus, tint, brightness), can 
also be changed. 

30 A visual user interface according to this aspect of the 

invention can overcome, for example, the above -identified 
deficiencies in previous video camera operation that are 
associated with the limited field of view of a video camera. 
For instance, a visual user interface according to the 

3 5 invention in which the target scene provides context for the 
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control scene can facilitate perception by a video camera _ 
operator of a current filmed scene in the context of the 
current filmed scene's surroundings. A camera operator's 
ability to engage in critical and sophisticated camera work 
5 with great precision is greatly facilitated. The visual user 
interface according to this aspect of the invention can 
enable" such" difficult camera work to be performed by 
relatively unskilled camera operators who heretofore would 
not have been able to perform such camera work acceptably. 

10 In accordance with another aspect of the invention, a 

visual user interface embodied in a single device enables 
control of the interaction of a device with a spatial region 
and the capability to make annotations regarding the 
interaction of the device with the spatial region. The 

15 visual user interface according to this aspect of the 
invention enables such annotations to easily be made 
contemporaneously with control of the device, so that a 
single person can perform both functions well. Additionally, 
the visual user interface can be implemented in a way (e.g., 

20 with a touchscreen and stylus) that enables the user to 

handwrj-te the annotations, thus facilitating a wide range of 
expression in creating annotations. Such a visual user 
interface also avoids the problems associated with voice 
annotation systems, e.g., disruption of the event or scene 

25 being filmed, and difficulty in searching through the 
annotations . 

The invention can be used in a wide variety of 
applications. In particular, the invention is useful in 
applications which require the control of a video device. 

3 0 Examples of such applications with which the invention can be 
used include movie and television film production and 
display, security camera systems and video conferencing 
systems. The invention can be used, however, in applications 
other than those in which a video device is controlled. For 

3 5 example, the invention could be used in controlling a 
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lighting device or devices so that a spatial region is 
illuminated in a desired way. 

RRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a simplified representation of a system with 
5 accordance with the invention. 

FIG. 2 illustrates a visual user interface, in - 
accordance with one embodiment of an aspect of the invention 
in which a visual user interface is used to control the 
interaction of a device with a spatial region, that is used 
10 to control the operation of a video device. 

FIG. 3 is a panoramic scene within which a sequence of 
scenes is to be filmed or displayed. 

FIGS. 4A through 4C illustrate successive filmed scenes 
from the panoramic scene of FIG. ^3 . 
15 FIGS. 5A through 5C each illustrate a visual user 

interface in accordance with the invention at the time that 
the corresponding one of the scenes of FIGS. 4A through 4C is 
being filmed, 

FIGS. 6A through 6C illustrate the operation of the 
20 visual user interface of FIG. 2 in accordance with an 

embodiment of the invention in which the area of view of the 
target scene can move as necessary to keep the control scene 
at least partly surrounded by the target scene. 

FIGS. 7A through 7E illustrate the use of a stylus and 
25 touchscreen to make spatial changes in a control scene that 
effect corresponding spatial changes in a filmed scene. 

FIGS. 8A and 8B illustrate a visual user interface in 
accordance with an embodiment of the invention in which the 
current filmed scene is displayed in addition to the control 
3 0 scene and the target scene. 

FIG. 9 illustrates a visual user interface in accordance 
with another aspect of the invention in which a visual user 
interface includes a space in which a user can make 
annotations regarding the device interaction with the spatial 
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region . 

DETAILED DESCRIPTION OF THE INVENTION 

Stated generally, a visual user interface according to 
one aspect of the invention can enable control of the 
5 interaction of a device with a spatial region. (Herein, such 
"interaction'- can also include the -display of an- interaction 
of a device with a spatial region.) The visual user 
interface includes a display of a control space and a target 
space. Typically, the target space has a specified 

10 relationship to the spatial region, e.g., the target space 

can represent some or all of the spatial region. The control 
space and target space can be displayed together so that at 
least a portion of the control space overlaps at least a 
portion of the target space, the content of the overlapping 

15 portions of the control space and target space being 

substantially the same, such that the target space provides 
context for the overlapping portion of the control space. 
The device is controlled in response to the state of the 
display of the control space. For example, the spatial 

20 relationship between the control space and the target space 
can be determined and used to control the device in a 
particular manner. The visual user interface enables a user 
to change the state of the display of the control space, 
thereby effecting control of the device and its interaction 

25 with the spatial region. 

A visual user interface in accordance with this aspect 
of the invention can be used, for example, to control a video 
device. Herein, "video device" encompasses any device for 
acquiring video data and/or generating a display from video 

30 data (e.g., a video camera, a video-enabled motion picture 

camera, a television, a computer display monitor, and a video 
or graphics projection system) . Hereafter, often, the 
invention is described in detail with respect to embodiments 
of the invention that can be used with video devices. It is 
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understood, however, that the invention has broader 
applicability and can be used with other types of devices, 
some of which are also discussed below. 

FIG. 1 is a simplified representation of a system 100 
5 with which the invention can be used. A video camera 101 is 
mounted on a movable camera mount of a tripod 102 (or other 
camera support device) . The movable camera mount can be 
embodied by any such mount (of which a number of known types 
exist) that enables desired movement (rotational, 
10 translational or both) of the video camera 101. Often, 

movable camera mounts are implemented to enable one or more 
of three types of rotational movement of a video camera: 
panning (i.e., rotation about a panning axis that changes the 
filming direction of a video camera in a right or left 
15 direction relative to a "normal" orientation of the video 
camera), tilting (i.e., rotation about a tilt axis that 
changes the filming direction of a video camera in an up or 
down direction relative to a "normal" orientation of the 
video camera) and rotating (i.e., rotation about a rotational 
20 axis that leaves the filming direction of a video camera 
unchanged or substantially unchanged and, instead, causes 
rotation of a filmed scene) . Advantageously, the movable 
camera mount can enable movement of the video camera 101 
having three rotational degrees of freedom (i.e., rotation 
25 about the pan, tilt and rotational axes) , and the invention 
is often described below with respect to embodiments that 
enable such movement. However, the invention can also be 
used with systems in which the video camera movement has only 
one or two rotational degrees of freedom. The invention can 
30 also be used with systems that enable translational motion 

(one, two or three degrees of freedom) of the video camera in 
addition to, or instead of, rotational movement. (For 
example, a particularly robust system according to the 
invention could be implemented to allow video camera motion 
35 having three rotational degrees of freedom and three 



BIMSDOCID: <WO 9904S57A1_L> 



wo 99/04557 



PCT/US98/14787 



- 10 - 

transiational degrees of freedom. ) As explained in more 
detail below, metrics are established for the movable camera 
mount that correlate positions of the movable camera mount 
with filming directions, orientations and/or positions of the 
5 video camera 101. (A filming direction can be specified in 
any appropriate manner, such as, for example, by specifying 
the content at the location of the -center of^ the- area being 
filmed by the video camera.) 

A user interface device 104, which can be any 

10 appropriate device that can display a visual user interface 
according to the invention, is used by a user to input 
instructions for use in controlling operation of the video 
camera 101. The user interface device 104 can be embodied 
by, for example, a conventional portable computing device, 

15 such as a notebook computer, subnotebook computer, personal 
digital assistant (PDA) or other similar device, together 
with, as appropriate, one or more associated user input 
devices, such as, for example, a mouse, keyboard, trackball 
or stylus. Embodying the user interface device 104 in a 

20 portable device is advantageous because it enables the user 
interface device 104 to be easily moved to filming locations 
together with the video camera 101 and tripod 102. As 
illustrated in FIG. 1, the user interface device 104 includes 
a portable computing device 105, as well as a stylus 106 thar 

2 5 can be used to contact a touchscreen (which can be used to 
display the visual user interface according to the invention) 
of the portable computing device 105 to effect input of 
instructions from the user to the portable computing 
device 105. However, the user interface 104 can be 

30 implemented by other devices, including computing devices 
that are not portable, such as, for example, a desktop or 
workstation computer. 

A data processing and storage device 10 3 is 
interconnected with the user interface device 104, the 

35 tripod 102 (in particular, a position encoder or encoders of 
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the movable camera mount) and the video camera 101 to enable 
communication therebetween. Such communication can be 
implemented using any appropriate methods and apparatus 
(e.g., serial or parallel digital communications apparatus 
5 and protocols, such as the RS232/C. protocol and associated 
apparatus), as known to those skilled in the art. Moreover, 
such communication can be either wired* or wireless , thus 
enabling the data processing and storage device 103 and the 
user interface device 104 to be implemented either as 

10 tethered (wired) or untethered (wireless) devices. A system 
in which the data processing and storage device 10 3 and the 
user interface device 104 are untethered can be advantageous 
because it affords greater freedom of movement for a user 
during use of those devices (and, in particular, during use 

15 of a visual user interface according to the invention) . 

The daca processing and storage device 103 can, for 
example, store video data acquired by the video camera 101 
and data describing the video camera metrics discussed above, 
as well as any other data and/or instructions (e.g., audio 

20 data and instructions for use of audio data) that are used in 
the operation of the system 100. The device 103 can also 
process instructions and data received from the video 
camera 101, the position encoder or encoders of the movable 
camera mount of the tripod 102 and user interface device 104, 

25 as well as transmit instructions and data to those devices. 
For example, when an instruction regarding the filming 
direction, position and/or orientation of the video 
camera 101 is received from the user interface device 104, 
the device 103 uses the metric information stored therein to 

30 determine the appropriate position of the movable camera 
mount and send an instruction to cause the movable camera 
mount to move to that position. Or, for example, the 
device 103 can receive video data representing filmed scenes 
acquired by the video camera 101, process and/or store the 

35 video data as necessary or desirable, and communicate data 
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and instructions to the user interface device 104 to cause 
generation of a display of a visual user interface according 
to the invention, as described elsewhere herein. The 
device 103 can also store data representing the camera 
5 metrics for each filmed scene acquired by the video 
camera 101. 

The" device 10 3 "can be implemented, for example, by a 
conventional digital computer, in particular, the device 103 
can advantageously be embodied in a portable computing device 

10 (examples of which are given above with respect to the 

description of the user interface device 104). As with the 
user interface device 104, embodying the data processing and 
storage device 10 3 in a portable device is advantageous 
because it enables the data processing and storage device 10 3 

15 to- be easily moved with the video camera 101 and tripod 102 
to filming locations. However, like the user interface 
device 104, the data processing and storage device 103 can be 
embodied by devices that are not portable, such as, for 
example, a desktop or workstation computer. 

20 The data processing and storage device 103 and the user 

interface device 104 can advantageously be embodied in the 
same device, thus reducing the number of separate pieces of 
apparatus necessary to implement the system 100. 

FIG. 2 illustrates a visual user interface 200, in 

25 accordance with one embodiment of this aspect of the 

invention, that is used to control the operation of a video 
device, such as the video camera 101 of FIG. 1. (The visual 
user interface 200 can also be used in controlling other 
types of devices, as discussed further elsewhere herein.) 

30 The visual user interface 200 includes a display of a target 
scene 201 and a display of a control scene 202. As 
illustrated in FIG. 2, the control scene 202 is a defined 
region within the target scene 201. The content of the 
control scene 202 is the same or substantially the same as 

3 5 the content of the target scene 201 that would otherwise be 
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present at the location of the control scene 202, such that 
the content of the target scene 201 provides context for the 
control scene 202. (This can be better appreciated in the 
illustration of a visual user interface according to the 
5 invention shown in FIGS. 5A through 5C, described below.) 
The boundaries of the control scene 202 can be delineated in 
any appropriate graphical manner as known by those "skilled^ in 
the art of implementing video displays. 



10 device in accordance with the state of the display of the 
control scene 202. As described in more detail below, the 
user can change the display of the control scene 202 to cause 
changes in the filmed scene. For example, if the visual user 
interface 200 is implemented in the user interface device 104 

15 (FIG. 1), changing the position of the control scene 202 
within the target scene 201 can cause an instruction to be 
sent to the position encoder or encoders of the movable 
camera mount of the tripod 102 to cause an appropriate change 
in the filming direction and/or position of the video 

20 camera 101 . 

Herein, a "filmed scene" is a scene that is being, has 
been, or can be filmed or displayed by a video device. As 
used herein, filmed scene typically refers to a 
representation (e.g., a frame of video data) of a part of a 

25 spatial region (which can be the entire spatial region) , 

though the term can also be used to refer to the part of the 
spatial region itself. A "control scene" is a displayed 
scene that is displayed as part of a visual user interface 
according to the invention and can be changed by a user to 

30 effect control of a device. (Similarly, a "control space" is 
a displayed space that is displayed as part of a visual user 
interface according to the invention and can be changed by a 
user to effect control of a device.) When the device is a 
video device, changing the control scene causes a change in 

3 5 the filmed scene. In such an embodiment, often, the content 



A filmed scene is acquired or displayed by the video 
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of the control scene is the same (or substantially the same) 
as the content of a current filmed scene, though, as 
described below, this need not necessarily be the case. A 
"target scene" is a displayed scene that is displayed as part 
5 of a visual user interface according to the invention and 
provides context for a control scene and/or is related to a 
control scene to effect particular operation of the 'device 
being controlled. (Similarly, a "target space" is a 
displayed space that is displayed as part of a visual user 

10 interface according to the invention and provides context for 
a control space and/or is related to a control space to 
effect particular operation of the device being controlled.) 
The target scene typically has a specified relationship to 
the spatial region with which the controlled device 

15 interacts, and, often, is a representation of some or all of 
the spatial region. Thus, when the controlled device is a 
video device, typically, the content of a filmed scene is at 
least partially (and, often, entirely) drawn from the target 
scene. Often, a target scene represents a "panoramic scene," 

20 which, as used herein, refers to a scene that can entirely 
encompass a filmed scene. In practice, a panoramic scene may 
be much larger than a filmed scene. For example, the target 
scene, may represent a panoramic scene which encompasses the 
entire field of view of a camera operator, or a panoramic 

25 scene which encompasses a 360 degree field of view from a 
specified location. When the target scene represents a 
panoramic scene, the target scene can be a "composite scene," 
which is a scene that is constructed from a combination of 
multiple filmed scenes. Below, particular aspects of the 

3 0 invention are described with respect to an implementation of 
the invention in which the target scene is a composite scene 
that represents a panoramic scene, and the control scene is 
the same (or nearly the same) as a current filmed scene (see, 
for example, FIGS. 5A through SB). 

35 FIG. 3 is a panoramic scene within which a sequence of 
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scenes ("filmed scenes," in accordance with the above - 
definitions) can be filmed or displayed. To illustrate the 
advantages of the visual user interface according to the 
aspect of the invention now being described, it will be 
5 assumed that a camera operator wishes to film the panoramic 
scene of FIG. 3 by beginning filming near the upper left 
corner of the panoramic scene, then sweeping downward to the 
center of the panoramic scene, and finishing by sweeping 
upward to the upper right corner of the panoramic scene. It 

10 is to be understood that the principles of the invention 
illustrated in the discussion below are also applicable to 
use of a visual user interface according to the invention to 
display a sequence of filmed scenes generated from a set of 
video data that represents the panoramic scene of FIG. 3, 

15 FIGS. 4A through 4C illustrate successive filmed scenes 

from the panoramic scene of FIG. 3 obtained during filming as 
described above. In previous manual control of a video 
camera, as described above, at any particular time the camera 
operator can look at either the filmed scene or the panoramic 

20 scene, but not at both simultaneously. Such video camera 
control is disadvantageous, as described in more detail 
above. For example, if the camera operator chooses to look 
at the filmed scene while filming, then, while the camera 
operator is filming the scene of FIG. 4A, for example, the 

25 camera operator sees only that scene. The camera operator 
must rely on his memory of the entire panoramic scene of 
FIG. 3 to know how to move the camera to get to the scene of 
FIG, 4B in a desired way (e.g., with a particular sweeping 
arc). Moreover, the camera operator must rely on such memory 

30 to recall the content of the scene of FIG. 4B itself, i.e., 
to know where he wants the camera to go. As can be 
appreciated, this can be a daunting task, particularly if the 
camera operator must film for an extended period of time 
without a break (i.e., without looking away from the filmed 

35 scene) . 
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FIGS, 5A through 5C each illustrate a visual user 
interface 500 according to the invention at the time that the 
corresponding one of the filmed scenes of FIGS. 4A through 4C 
is being filmed. Like the visual user interface 200 of 
5 FIG. 2, the visual user interface 500 includes a display of a 
control scene 502 that is a defined region within a display 
of~a target scene 501, such that the content 'of the target ^ 
scene 501 provides context for the control scene 502, 



10 interface 500, the control scene 502 is the same as the 
filmed scene (examples of which are shown in FIGS. 4A 
through 4C) and the target scene 501 is the same as the 
panoramic scene (shown in FIG. 3), Thus, a user of the 
visual user interface 500 can directly (and viscerally) 

15 control the state of the filmed scene at any given time by 
controlling the display of the control scene 502. For 
example, as illustrated in FIGS. 5A through 5C, the control 
scene 502 (and, thus, the filmed scene) can be moved to 
various locations within the target scene 501. In accordance 

20 with the aspect of the invention now being described, then, 
during filming of the panoramic scene of FIG. 3 as described 
above, the camera operator would look at the visual user 
interface 500 illustrated in FIGS. 5A through 5C, rather than 
through the viewfinder of the camera (direct manual camera 

25 control) or at a display of the filmed scene (remote manual 
camera control), as is depicted in FIGS. 4A through 4C, The 
benefits of the visual user interface 500 are readily 
apparent. As the camera operator moves from filmed scene to 
filmed scene by changing the control scene 502, the current 

30 filmed scene (displayed as the control scene 502) can always 
be simultaneously viewed in context of the remainder of the 
panoramic scene (which is displayed as the target scene 501) . 

In addition to being used to control operation of a 
video camera, the visual user interface 500 could also be 

3 5 used to control the display of an existing set of video data 



As shown in FIGS. 5A through 5C, in the visual 



^NSDCXID: <WO. 



.99(MS57A1_L> 





WO -^^4557 « 



PCT/US98/14787 



- 17 - 



that represents a panoramic scene. Again, the user would 
make changes in the control scene 502, such changes causing 
corresponding changes in the displayed filmed scene. 
While, as indicated above and by the detailed 
5 description of the invention below, the invention is 

particularly useful as applied to control a video device, the 
invention is not ~ limited" to such^ app"lications . ~ Tn " general'; a ' ~ 
visual user interface according to the invention can be used 
to control the interaction of any type of device with a 
10 spatial region and can control the device in any manner or to 
perform any function, such as, for example, panning, tilting, 
rotating, zooming, focusing, aiming, shooting, firing, 
tracking, projecting, pointing, pasting and painting. 



15 invention, such as the visual user interfaces 200 (FIG. 2) 
and 500 (FIGS, 5A through 5C) , could also be used to control 
a lighting device or devices so that a spatial region is 
illuminated in a desired way. A user could, for example, 
position a control scene within a target scene to cause a 

20 spotlight to be directed to a particular location within a 
spatial region represented by the target scene. Making the 
control scene larger or smaller could, for instance, cause 
the emitted light to become focused on a correspondingly 
larger or smaller area. Changing the focus of the control 

25 scene might, for example, increase or decrease the 
collimation of the emitted light, while changing the 
brightness of the control scene could cause corresponding 
changes in the intensity of the emitted light. The above - 
described techniques for using a visual user interface 

30 according to the invention to control a lighting device are 
merely illustrative; they are neither necessary nor 
exhaustive . 

In general, a control space displayed as part of a 
visual user interface according to the invention can be 
3 5 changed in any manner. For example, a user can cause a 



For example, a visual user interface according to the 
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display of a control space (e.g., a control scene) to chaage 
spatially with respect to a display of a target space (e.g., 
a target scene) , such change causing a predetermined change 
in the operation of a device (e.g., a filmed scene of a video 
5 device). As one illustration of such spatial changes, a user 
can cause the display of a control scene to change in size, 
shape, angular orientation and/or location with respect to 
the display of a target scene. As used to control operation 
of a video camera (or other video device), for example, such 

10 changing of the control scene can be manifested as, for 

example, panning, tilting, rotating and/or zooming. A user 
can also change the display of a control space in other ways. 
For example, a user can change visual parameters (such as 
focus, tint, contrast and/or brightness) of the display of a 

15 control scene. As used to control operation of a video - 
camera or video enabled motion picture camera, for example, 
such changing of the control scene can be manifested as, for 
example, changes in the lens focusing mechanism, white 
balance, pedestal, and/or aperture and/or gain, respectively, 

20 of the camera. 

A control space (e.g., control scene) can, in general, 
have any predetermined relationship to the device interaction 
with the spatial region (e.g., filmed scene). For example, 
in a visual user interface according to the invention that is 

25 used to control a video device, it is anticipated that it 
will often be advantageous to have a control scene that is 
the same (or substantially the same) as the filmed scene. 
However, this need not necessarily be the case. The control 
scene can be larger or smaller than the filmed scene. For 

30 example, it may be desirable to make the control scene 
proportionally larger or smaller than the filmed scene. 

Further, in general, the interaction of the device with 
the spatial region (e.g., filmed scene) can change in 
response to changes in the display of the control space 

35 (e.g., control scene) in any predetermined manner. For 
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example, in a visual user interface according to the 
invention that is used to control a video device, it is 
anticipated that it will often be advantageous for the filmed 
scene to change in the same manner as the control scene, 
5 e.g., if the control scene is changed in size, the filmed 
scene is changed in size in the same proportions (and, 
perhaps, by the same amount or amounts). However, again, - 
this need not necessarily be the case. 

As indicated above, in a visual user interface according 

10 to the invention that is used to control a video device, the 
filmed scene can advantageously be the same or substantially 
the same as the control scene (as in the visual user 
interface 500 described above), and can advantageously change 
in a manner that directly corresponds to changes in the 

15 control scene. Such a visual user interface is advantageous, 
for example, because it provides the user with a visceral 
sense of the control being effected. 

Additionally, the target space (e.g., target scene) can, 
in general, have any predetermined relationship to the 

20 control space (e.g., control scene) . For example, in a 

visual user interface according to the invention that is used 
to control a video device, it is anticipated that it will 
often be advantageous for the control scene to be encompassed 
(typically entirely, but at least partially) within the 

25 target scene. This is advantageous because it provides 
context to the user that can aid the user in changing the 
control scene in a manner that provides desired control of 
the filmed scene. Again, though, there need not necessarily 
be such a relationship between the control scene and target 

30 scene. 

In general, both the display of the control space and 
the display of the target space can be either a static 
display or a dynamic display. It is anticipated that, in 
many applications of a visual user interface according to the 
3 5 invention, it will be advantageous to have a dynamic control 
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space display. When a visual user interface according to J:he 
invention is used to control a video device, for example, 
this can usually be easily accomplished by making the control 
scene the same as the filmed scene, which, typically, will be 
5 dynamic. Additionally, in a visual user interface according 
to the invention that is used to control a video device, 
whether ^a' static or dynamic targe't scene is used will 
typically involve a tradeoff between the cost of generating 
the display of the target scene (static is typically cheaper) 

10 and the quality of the display of the target scene (dynamic 
generally provides a more accurate display) . 

Above, a visual user interface, according to one aspect 
of the invention, that enables control of the interaction of 
a device with a spatial region has been described generally. 

15 Below, a detailed description is given of ways in which a 
visual user interface according to this aspect of the 
invention can be implemented. The description is generally 
made with respect to a visual user interface that is similar 
to the visual user interfaces 200 (FIG. 2) and 500 (FIGS. 5A 

20 through 5C), i.e., a visual user interface including i) a 
targen scene that represents a panoramic scene, and ii) a 
control scene that is smaller than, and subsumed within, the 
target scene, and that is, when the visual user interface is 
used to control a video device, the same (or nearly the same) 

25 as a filmed scene. 

To enable the spatial characteristics (e.g., location 
within the target scene, and angular orientation, size and 
shape relative to the target scene) of the control scene to 
be used to control operation of a video device, metrics must 

30 be recorded that relate particular content in the target 

scene to particular physical attributes of the device being 
controlled. For example, if the device being controlled is a 
video camera, then, for a particular camera location relative 
to the panoramic scene, the location of a region of content 

35 within the target scene (i.e., the filming direction) can be 



BNSDOCID: <WO 9904557A1_L> 




wo ^9^04557 - 



PCT/US98/14787 



- 21 - 



related to a mechanical position of the video camera (e.g., 
the angular orientation about the pan and tilt axes of a 
movable camera mount of a tripod-mounted video camera) . 
Additionally, the angular orientation of a region of content 
5 can be related to a mechanical position of the video camera 
(e.g., the angular orientation about the rotational axis of a 
movable camera mount of a tripod-mounted video camerah. - - 
Further, the size of a region of content can be related to 
the camera lens characteristics (e.g., the setting of a zoom 

10 lens) . It may also be possible to relate the shape of a 
region of content to camera characteristics: for example, 
the aspect ratio of a rectangular region of content can be 
related to the aspect ratio of the imaging device of the 
video camera to appropriately subsample the filmed scene 

15 recorded by the camera. The determination of such video 
camera metrics can be accomplished using any appropriate 
method. For example, one such method is described in U,S, 
Patent No. 4,847,543, the disclosure of which is incorporated 
by reference herein. As indicated above (see the description 

20 of FIG. 1), the video camera metrics (i.e., encoded positions 
of the video camera) can be stored in a data storage device 
and used to control a position encoder or encoders of a 
movable camera mount to appropriately position the video 
camera in response to spatial changes in a control scene made 

25 by a user of a visual user interface according to the 
invention. 

If, instead, the video device being controlled is a 
video display device, then the location, angular orientation, 
size and shape of a region of content (i.e., the content of 

30 the control scene) within the target scene can be used to 

subsample video data that is used to generate the display of 
the target scene. In many cases, the video data used to 
generate the target scene will have previously been acquired 
and stored on an appropriate data storage device. However, 

35 it is also possible to subsample the target scene video data 
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(in accordance with the control scene content) as it is being 
acquired to effect control of a displayed filmed scene. 
Methods for accomplishing such subsampling are described in, 
for example, U.S. Patent Nos, 5,185,667 and 5,444,478, the 
5 disclosures of which are incorporated by reference herein. 

Often, a target scene that represents a panoramic scene 
is a composite scene, i.e., a scene that is "constructed from 
a combination of multiple filmed scenes. To form the 
composite scene, a panoramic scene that is to constitute the 

10 target scene can be scanned by a video camera that records 

filmed scenes (i.e., video "frames") at the frame rate of the 
video camera (e.g. , 30 frames per second) . (The filmed 
scenes could also be acquired by scanning the panoramic scene 
or defined parts of the panoramic scene with multiple video 

15 cameras.) The filmed scenes and/or parts of filmed scenes 
are then combined in some manner to produce the composite 
scene. The filmed scenes are often acquired as analog data 
and the method of combining filmed scenes operates on digital 
data, thus necessitating analog to digital conversion of the 

20 data. This can be accomplished, as known by those skilled in 
the art, using conventional methods and apparatus. The video 
camera or cameras used to acquire the filmed scenes from 
which the composite scene is to be created can be panned, 
tilted, rotated and zoomed to enable the full resolution of 

25 the camera or cameras to be used in a variable and cumulative 
manner, thereby enabling the composite scene to be 
constructed from multiple filmed scenes which each have up to 
the full resolution of the camera or cameras. 



30 together to form the composite scene. Each filmed scene 

represents a portion of the panoramic scene. The content of 
the filmed scenes can, and typically will, overlap. 
Stitching the filmed scenes together causes the content of 
the filmed scenes to be pieced together so that corresponding 

3 5 overlapping parts of filmed scenes are blended together to 



For example, the filmed scenes can be "stitched 
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make Che composite scene appear to be (relatively) seamless. 
Stitching together of multiple filmed scenes to form a 
composite scene can be accomplished using any appropriate 
method. Since methods that produce the highest quality 
5 composite scenes are usually the most computationally 

expensive, the particular method used may be chosen based on 
a tradeoff between computational cost and quality of the 
composite scene. Frame stitching methods that can be used 
with the invention are described in, for example, U.S. Patent 
10 Nos. 5.396,583, 5,510,830 and 5,262,856, the disclosures of 
which are incorporated by reference herein. 

A composite scene can be formed from multiple filmed 
scenes in any other appropriate manner, as well. For 
example, the video camera metrics can be used to locate each 
15 filmed scene within the panoramic scene. Filmed scenes can 
then be selected, based upon their location, and pieced 
nogecher like a puzzle so that each selected filmed scene 
adjoins or overlaps other adjacent selected filmed scenes. 
However, the matching at the edges of the filmed scenes may 
20 not be exact, so that a composite scene formed in this way 
may not appear as seamless as a composite scene formed by 
stitching. 

For many applications with which the invention can be 
used, it is preferable that the filmed scenes from which a 

25 composite target scene is constructed be acquired prior to 
use of the visual user interface to control a device, since, 
as indicated elsewhere herein, the target scene is often used 
to provide context for the control scene. This can be done, 
for example, by using a video camera in a conventional manner 

30 to film a panoramic scene. Or, it may be possible to use the 
control scene to operate a video camera to acquire filmed 
scenes from which a target scene can be constructed. In 
particular, this can be accomplished if the content of the 
control scene is the same as that of the filmed scene. The 

35 absence of the target scene is not greatly detrimental, since 
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the user is only seeking to sweep the video camera over all 
of the content of the panoramic scene, rather than attempting 
to perform precision camera work. 

However, it need not be the case that the filmed scenes 
5 from which a composite target scene is constructed are 

acquired prior to use of the visual user interface. If the 
visual user interface is being used to control a video 
camera, the composite scene can be constructed "on the fly" 
from filmed scenes as the visual user interface is used to 

10 control the video camera to acquire the filmed scenes. 

Further, when a visual user interface according to the 
invention is used to control a video camera, in either an 
implementation in which the composite scene is constructed on 
the fly or an implementation in which the composite scene is 

15 constructed before use of the visual user interface, the 
content of the composite scene can be updated with newly 
acquired filmed scenes . Such updating can be done 
selectively to reduce the computational burden associated 
therewith. For example, only every nth filmed scene might be 

20 used to update the composite scene. Or, a record could be 
maintained of the time of the most recent update of each 
defined part of the composite scene, and only filmed scenes 
corresponding to "stale" parts of the composite scene (i.e., 
parts of the composite scene that have not been updated for 

25 greater than a predetermined duration of time) used to update 
the composite scene. Additionally or alternatively, "stale" 
parts of the composite scene can be indicated by a graphical 
effect which may also be implemented so as to indicate the 
relative "staleness" of the particular part of the composite 

30 scene (i.e., the magnitude of the duration of time that the 
part of the composite scene has not been updated) . For 
example, the color of a "stale" part of the composite scene 
can be gradually faded to black and white over time, the 
boldness of the color of that part of the composite scene 

35 indicating its relative "staleness." 
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In a visual user interface such as the visual user - 
interfaces 200 (FIG. 2) and 500 (FIGS. 5A through 5C), the 
display of the target scene must be continually adjusted as 
the display of the control scene is moved to different 
5 locations within the target scene. The boundary of the 

control scene can be changed using conventional GUI methods, 
as discussed further below. The location of the boundary of 
the control scene within the target scene at any time can be 
determined from those methods. Appropriate video data can 
10 then be selected from the stored video data representing the 
entire composite target scene so that a display of the target 
scene is generated in all of the area of the visual user 
interface that is allotted for the display of the target 
scene and that is not being used by the display of the 
15 control scene. 

The control scene in a visual user interface according 
to the invention need not always be confined to the 
boundaries of the target scene. However, since the use of a 
target scene to provide context for a control scene is 
20 particularly advantageous, it may be desirable, where the 

target scene is related to the control scene so as to provide 
such context, to at least confine movement of the control 
scene relative to the target scene so that the control scene 
is always at least partly surrounded by the target scene. 
25 There are two possible scenarios that can be addressed. In 
one, the display of the target scene is implemented so that 
the area of view encompassed by the target scene remains 
constant. In such an implementation, limitation of movement 
of the control scene as described above can be readily 
3 0 accomplished by those skilled in the art of implementing 
video displays. In the second scenario, the display of the 
target scene is implemented so that the area of view 
encompassed by the target scene can move. This scenario may 
be useful in situations where it is desired to display a 
3 5 target scene that encompasses only part of a panoramic view 
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to be filmed (perhaps because it is felt that the size of -the 
target scene is sufficiently small that displaying the entire 
panoramic scene as the target scene will cause features in 
the target scene to be too small to be easily discerned) . In 
5 this case, the visual user interface could be implemented so 
that when the control scene reaches a predefined location 
near a boundary of the target scene (referred to hereafter as 
"a predefined distance from a boundary of the target scene," 
though it is possible to implement this embodiment of the 

10 invention so that part of the control scene may already be 

extending outside of the boundary of the target scene at this 
point) , the area of view of the target scene moves as 
necessary' to maintain that predefined distance from the 
boundary of the target scene. The above may be useful to 

15 better match the control scene and target scene sizes and 
resolutions to the display type used. 

FIGS. 6A through 6C illustrate operation of the visual 
user interface 200 (FIG. 2) in accordance with an embodiment 
of the invention in which the area of view of the target 

20 scene 201 can move as necessary to keep the control scene 202 
at least partly surrounded by the target scene 201. In 
FIGS. 6A through 6C , for aid in illustrating this embodiment 
of the invention, a panoramic scene 601 is shown superimposed 
over the visual user interface 200. As' can be seen, the 

25 target scene 201 can include only part of the panoramic 

scene 601 at any one time. In FIG. 6A, the target scene 201 
includes content near the center of the panoramic scene 601, 
and the control scene 202 is positioned within the target 
scene 201 near the bottom of the target scene 202. It is 

30 assumed that the visual user interface 200 is implemented to 
allow no portion of the control scene 201 to come closer than 
a predetermined minimum distance from the boundary of the 
target scene 201, and that the distance 602 is greater than 
this minimum distance. In FIG. 6B, the control scene 202 has 

3 5 been moved up and to the left with respect to the target 
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scene 201 so that the left boundary of the control scene- 202 
is a distance 603 from the left boundary of the target 
scene 201. The distance 603 is assumed to be equal to the 
minimum distance; therefore, the content of the target 
5 scene 201 is the same as in FIG, 6A. in FIG, 6C , the control 
scene has been moved even further up and to the left with 
respect to the target scene 201. In order to maintain the 
minimum distance .between the boundary of the control 
scene 202 and the boundary of the target scene 201, the 
10 target scene 201 moves to the left - in coordination with the 
movement of the control scene 202 - with respect to the 
panoramic scene (in the visual user interface 200, this would 
actually appear as the panoramic scene 601 moving to the 
right in the target scene 201) . The distance between the 
15 left boundaries of the control scene 202 and target scene 201 
is still 603. Further, the distance 604 between the upper 
boundaries of the control scene 202 and target scene 201 is 
greater than the minimum distance (i.e., greater than the 
distance 603); thus, the target scene 201 does not move with 
20 respect to the panoramic scene 601 in that direction. 

Detection of the distance between the boundaries of the 
control scene 202 and target scene 201 can be accomplished 
using conventional GUI techniques. The content of the -target 
scene 201 can modified by appropriately stitching additional 
25 filmed scenes to a previously existing target scene 201 in 
the same manner as described elsewhere herein for creating a 
composite target scene. 

As indicated above, both the display of a target scene 
and the display of a control scene of a visual user interface 
30 according to the invention can be either dynamic or static. 
For many applications (e.g., videotaping with a video 
camera) , it is desirable that the display of the control 
scene be dynamic, since the content of the control scene 
corresponds exactly or very nearly to the content of the 
35 current filmed scene, and it is necessary or desirable to 
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have as accurate a depiction of the control scene as possi-ble 
to ensure that the control scene will be controlled to obtain 
or produce the desired sequence of filmed scenes. In 
applications in which the control scene duplicates the 
5 current filmed scene, the control scene can be generated 
directly from the filmed scene (e,g,, from a display feed 
from a video camera) using conventional methods and 
apparatus . 

In some situations, it may be acceptable to have a 

10 static control scene display. When the target scene display 
is also static, the content of a static control scene display 
can easily be generated as the content that would otherwise 
be present at that location within the target scene. When 
the display of the control scene is static, the control scene 

15 can be demarcated in any appropriate manner, such as by, for 
example, displaying an outline of the area of the control 
scene or by making the area of the control scene more or less 
bright than the area of the target scene. The use of a 
static control scene display may be desirable because the 

20 generation of a static scene can typically be accomplished 
more easily and inexpensively than the generation of a 
dynamic scene. While a visual user interface including a 
static control scene display can be used for any application, 
such a visual user interface may be advantageously used, in 

25 particular, to control a lighting device, as described above, 
since the loss of fidelity associated with the static control 
scene display would typically not be detrimental in effecting 
accurate control of a lighting device. 

The display of the target scene can also be either 

30 dynamic or static, the choice usually depending, as described 
above, on the evaluation of a tradeoff between the quality of 
the target scene display and the cost of generating that 
display. A static target scene display can be formed as a 
composite scene as described above, e.g., using a frame 

35 stitching method as described above to create a composite 
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scene from multiple filmed scenes. 

A dynamic target scene display can be constructed from 
the sequences of filmed scenes acquired by multiple video 
cameras that each shoot a fixed region of space within a 
5 panoramic scene at the same time, the regions adjoining 

and/or overlapping each other. The operation of the multiple 
video cameras is synchronized, as Jcnown by those skilled in 
the art, so that frames of video data are acquired at the 
same time (or substantially the same time) by each of the 

10 video cameras. Such synchronization can be accomplished in 
any appropriate manner, such as supplying the cameras with a 
genlock reference signal and "free running" the SMPTE time 
code, as known by those skilled in the ari:. Filmed scenes 
(i.e., frames of video data) acquired at the same time from 

15 the multiple video cameras can be combined to form a dynamic 
composite scene. Once such a set of temporally coincident 
filmed scenes has been determined, the filmed scenes from the 
set can be combined to form a composite target scene by, for 
example, stitching together adjacent and/or overlapping 

20 filmed scenes in a manner similar to that described above for 
creating a static composite target scene. Frame stitching 
methods that can be used with the invention to create a 
dynamic composite scene are described in, for example, U.S. 
Patent Nos . 5,444,478 and 5,187,571, the disclosures of which 

25 are incorporated by reference herein. Alternatively, a 

dynamic composite target scene can be created by using the 
video camera metrics to locate each sequence of filmed scenes 
within the panoramic scene, and piecing together the 
sequences of filmed scenes, based upon their locations, so 

30 that the sequences of filmed scenes adjoin or overlap 

adjacent sequences of filmed scenes, in a manner similar to 
that described above for creating a static composite target 
scene . 

In some applications of the invention, such as filming 
35 of movies or television shows, the visual user interface can 
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be used to control a relatively expensive, high quality vijdeo 
camera. The use of multiple such cameras to acquire filmed 
scenes for use in constructing a dynamic composite target 
scene would, in many cases, be undesirably expensive. Thus, 
5 when it is desired that the target scene display be dynamic, 
the cost of producing such a dynamic target scene display can 
be reduced by using cheaper video cameras (e. g , , single-CGD 
cameras) to acquire the filmed scenes used to construct the 
composite target scene. An expensive camera can be used to 

10 acquire the filmed scenes (and, if the interface is so 

configured, the control scene) that are actually to comprise 
the desired video display. For example, the expensive camera 
can be mounted on a movable camera mount ''of a tripod and 
controlled by a position encoder or encoders as described 
'15 above, and the inexpensive cameras can be mounted at other 
locations on the tripod and positioned (i.e., aimed in 
particular directions) so that the filmed scenes obtained by 
the inexpensive cameras can be combined to produce the 
composite scene. Preferably, the nodal points about which 

20 the inexpensive cameras are positioned are as close to 

coincident as possible. Though the target scene display will 
be of lesser quality than the control scene display, this 
will often not be an important detriment. The fact that the 
target scene display is dynamic rather than static is 

25 advantageous in and of itself. 

It may also be possible to use a single relatively 
inexpensive camera (e.g., one single-CCD camera) including a 
very wide angle lens (e.g., a "fish-eye" lens) to acquire 
scenes having sufficient resolution to be used as a target 

30 scene. To produce an acceptable target scene display, it 
would typically be desirable to perform distortion 
correction, as known to those skilled in the art, on the 
filmed scenes (video data) acquired by such camera. The use 
of a camera as described above to acquire filmed scenes, 

35 including application of distortion correction to the filmed 
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scenes, is described in, for example, the above-mentioned 
U.S. Patent No. 5,185,667. 

Any appropriate apparatus and methods can be used to 
enable a user to make changes in a control scene. For 
5 example, a user can be enabled to make spatial changes in a 
control scene (e.g., changes in the size, shape, angular 
orientation and/or location of the control scene) by, for 
example, using an appropriate input device (e.g., a mouse, 
keyboard, trackball, stylus, joystick or Trackpoint'^^ 

10 mechanism) to appropriately change the boundary of the 
control scene in accordance with the requirements of 
conventional graphical user interface (GUI) software and 
hardware that implements such functionality {e.g., the 
hardware and software used to implement such functionality in 

15 GUI -based operating systems such as the Windows^^ operating 
systems). If, for example, the visual user interface is 
displayed on a touchscreen, the above -described changing of 
the size, shape, angular orientation and/or location of the 
control scene can be accomplished using a stylus or, perhaps, 

20 the user's finger. Other apparatus can also be used to make 
spatial changes in a control scene. For example, a 
conventional rocker switch can be used, as can be readily 
understood by those skilled in the art, to effect zoom 
control of a control scene, as well as control of the 

25 location of the control scene. Or, for example, a 

conventional rotating knob can be used, as can also be 
readily understood by those skilled in the art, to effect 
rotation of a control scene. Conventional pushbuttons can 
also be used to effect desired changes in the size, shape, 

30 angular orientation and/or location of the control scene. 
Many other types of apparatus can also be used to effect 
appropriate control of a control scene (including control of 
visual parameters of the control scene, as discussed 
elsewhere herein), as can readily be appreciated. For 

3 5 example, a voice recognition system can be used to enable a 
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user to speak commands (e.g., "Zoom", Pan Left", etc.) to _ 
effect control of a control scene. 

FIGS, 7A through 7E illustrate the use of a stylus and 
touchscreen to make changes in the location of a control 
5 scene display that effect corresponding spatial changes in a 
filmed scene. In FIG. 7A, the stylus 701 is positioned 
within the control scene 702 and dragged across the 
touchscreen to the right, causing the filmed scene to pan to 
the right. In FIG, 7B, the stylus 701 is positioned within 

10 the control scene 702 and dragged across the touchscreen in a 
downward direction, causing the filmed scene to tilt 
downward. In FIG. 7C, the stylus 701 is positioned within 
the control scene 7 02, dragged across the touchscreen to the 
right, then, transitioning gradually in a smooth arc, dragged 

15 across the touchscreen downward, causing the filmed scene to 
begin panning to the right, then, transitioning gradually in 
accordance with the arc made by the stylus 701, end by 
tilting downward. In FIG. 7D, the stylus 701 is positioned 
within the control scene 702 and dragged across the 

20 touchscreen upward and to the left, causing the filmed scene 
to pan to the left and tilt upward at the same time. In 
FIG. 7E, the stylus 701 is used to draw a circle on the 
touchscreen outside of the control scene 702, thus causing 
the control scene 7 02 to move from the location shown in 

25 FIG. 7E to a location defined by the circle (this can be 

implemented using a bounding box or "lasso" GUI tool, such as 
is implemented in the Adobe Photoshop^^ software) and thereby 
causing the filmed scene to move in a similar manner. (The 
same type of control could be effected by touching the stylus 

30 at a point on the touchscreen outside of the control 

scene 702.) As can be appreciated, many other similar types 
of stylus moves can be used to move the control scene 702 and 
thereby control the filmed scene. Further, as can be 
appreciated, when a display screen that is not a touchscreen 

35 is used to display the visual user interface, such control 
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moves could also be effected using, for example, a mouse., (or 
device having similar functionality, such as a trackball) to 
control the position of a pointer on the display screen. 

Though not illustrated in the drawings, the stylus 701 
5 could also be used to change the size of the control 

scene 702. (Other apparatus may be used for this purpose as 
well, such as one or more rocker switches or pushbuttons, as 
discussed above.) For example, conventional GUI techniques 
(such as are used in, for example, the Windows^" operating 
10 systems) could be implemented so that, when the stylus 701 is 
placed on the outline of the control scene 702, then dragged 
across the touchscreen, the outline of the control scene 702 
moves with the stylus, thus changing the size of the control 
scene 702. For example, moving the stylus 701 toward or away 
15 from the content of the control scene 702 could cause the 
control scene 702 to become smaller or larger, respectively, 
which could, in turn, cause the filmed scene to become 
smaller or larger, respectively (i.e., the stylus 701 is used 
to change the control scene 702 to effect zooming) . The 
20 control scene 702 can be made as small as desired, subject 

only to the limitation that the control scene 702 must remain 
large enough to enable control by the interface device used 
to change the control scene 702 and/or to the constraints 
imposed by a device (e.g., the lens of a video camera) being 
25 controlled. Similarly, the control scene 702 can be made as 
large as desired, subject only to the limitation imposed by 
the display screen on which the visual user interface is 
displayed and/or, again, to the constraints imposed by the 
video device . 

30 Changes other than spatial changes (e.g., changes in 

visual parameters such as focus, tint and/or brightness) can 
also be made in a control scene (which, in turn, effects 
corresponding control of the video device) using GUI 
techniques. For example, the ability to control visual 

35 parameters such as focus, tint and/or brightness could be 
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implemented using GUI slider bars that permit continuous 
adjustment between two extreme values. Tint and brightness 
changes can also be implemented using conventional mechanical 
or electronic controls that are usually already part of a 
5 display device. A display device for use in displaying a 
visual user interface according to the invention could also 
be modified to include- a mechanical or electronic control -for 
adjusting the focus of the control scene. A variety of other 
types of physical apparatus, including, for example, rocker 

10 switches, pushbuttons and knobs can also be used to effect 
control of visual parameters of a control scene. In general, 
any appropriate control mechanism can be used, as can be 
appreciated by those skilled in the art. 

In some situations, it may be desirable to modify the 

15 user control in a predetermined manner. For example, when 

the user moves the location of a control scene, such movement 
may be more shaky or jerky than desired. Conversely, it may 
be desirable - to create a desired visual am±)iance , for 
example - to make a movement of a control scene from one 

20 location to another more jerky than was the actual movement 
by the user. In accordance with a further aspect of the 
invention, user movement of the control scene (and, thus, 
movement of the filmed scene) can be modified according to a 
predetermined method. Such modification can be accomplished 

25 using known filtering techniques. For example, the jerkiness 
of movement of a control scene from one location to another 
can be increased or decreased using the Dynadraw technique 
developed by Paul Haeberli of Silicon Graphics, Incorporated, 
described, for example, at the World Wide Web address 

30 www.sgi.com/grafica/dyna/index.html (as of January 27, 1997) 
and included as part of the SGI Developers Toolbox CD issued 
in 1997. 

In a similar vein, when a movement of a control scene is 
not controlled directly by the user (e.g., when the user 
3 5 marks a location or area outside of a control scene to cause 
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the control scene to move to a new location) , a desired 
control scene movement can be produced in accordance with a 
predetermined method. The creation of a control scene 
movement from scratch is, typically, simpler than 
5 modification of a user movement, and can be readily 

accomplished by those skilled in the art using an appropriate 
mathematical model of the characteristics of the desired - 
physical movement . 



10 modifying or creating a movement of a control scene can be 
accomplished in any appropriate manner. For example, the 
user could select a method from a menu of choices, the menu 
being made available to the user using any of a variety of 
well -known techniques . 
15 A single visual user interface according to the 

invention could also be used to control multiple devices, 
e.g.r control the operation of multiple video cameras. 
Assuming that the orientation of each video camera with 
respect to the scene to be filmed is known, each camera 
20 orientation could be related to the orientation of a target 
scene of the visual user interface with respect to the scene 
to be filmed. Then, as a control scene is changed within the 
target scene, each of the video cameras is controlled as 
appropriate to film the region corresponding to the control 
25 scene. Alternatively, changes in the operation of the video 
cameras can be referenced to one of the video cameras, and 
that video camera controlled directly using the visual user 
interface. The visual user interface could also be 
implemented to include a display of the filmed scene from 
30 each of the video cameras, as well as a mechanism to enable 
specification of the video camera from which a filmed scene 
is to be acquired at any given time. Such multiple video 
camera control can be particularly useful, for example, in 
filming a live event such as a sports event, 
35 FIGS. 8A and SB illustrate the visual user interface 500 



The specification of the predetermined method for 
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(FIGS. 5A through 5C) , modified to include a display of a- 
current filmed scene 801 in addition to the displays of the. 
control scene 502 and target scene 501. As indicated above, 
the content of the control scene 502 can be the same as the 
5 content of the current filmed scene 801. If desired (and as 
illustrated in FIGS. 8A and SB), the display of the current 
filmed scene 801 can be implemented so that the current 
filmed scene 801 cannot be changed (e.g., moved, rotated, or 
made larger or smaller) by a user. (The current filmed scene 

10 801 need not necessarily be implemented in such fashion, 
however.) Thus, no matter how a user changes the control 
scene 502, the current filmed scene 801 provides the user a 
display of the content that is currently being filmed or 
displayed which remains constant in size, resolution, angular 

15 orientation and location within the visual user 

interface 500. This can be appreciated by comparing FIG. 8A 
to FIG. 8B, In FIG. 8B , the control scene 502 has been made 
smaller and shifted slightly down and to the left as compared 
to FIG. 8A. Provision of a stable display of the current 

20 filmed scene 801 can be useful, for example, when the user 
makes the control scene 502 so small within the target 
scene 501 that it is difficult to discern the content of the 
control scene 50 2 (and, thus, the filmed scene), when the 
user has rotated the control scene 502 a great deal, or when 

25 the user is moving or rotating the control scene 502 rapidly. 
FIG. 9 illustrates a visual user interface 900 in 
accordance with another aspect of the invention in which a 
visual user interface includes a space in which a^ user can 
make annotations regarding the device interaction with the 

30 spatial region (e.g., when the device is a video device, 
annotations regarding a filmed scene or scenes). For 
convenience in illustrating the principles of this aspect of 
the invention, the following description will be made with 
regard to an embodiment of the invention in which the visual 

35 user interface 900 is adapted for use in controlling a video 
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device . 



Similar to the visual user interface 500 as depicted in 
FIGS. 8A and 8B, the visual user interface 900 includes 
displays of a target scene 901. a control scene 902 and a 
5 current filmed scene 903. Each of the scenes 901, 902 

and 903 is displayed and operates as described above. (In 
particular, the scenes" 901, 902 arid 9 03 enable a user to - 
control the operation of the video device.) In addition to 
Che scenes 901, 902 and 903, the visual user interface 900 
10 includes an annotation space 904. 

The annotation space 904 enables a user to make textual 
nota-tions as the filmed scenes are being filmed or displayed 
by the video device. The user can, of course, make any 
desired textual notations. One advantageous use of the 
15 annotation space 904 is to make textual notations regarding 
the content of a filmed scene or scenes as those scenes 
occur . 

As illustrated in FIG. 9, a time scale is automatically 
displayed at the lefthand side of the annotation space 904. 
20 The presence of the time scale in the annotation space 904 
facilitates identification of the time at which annotations 
are made. The display of the time scale can be implemented 
so that the time scale appears to scroll through the 
annotation space 904 as time passes during filming or 
25 display. The display of the time scale can also be 

implemented so that the times in the time scale correspond to 
time codes (e.g., the well-known SMPTE time codes) associated 
with particular video data (e.g., a particular video frame or 
frames) . Time -identified annotations (even when the time 
30 associated with the annotation only approximately identifies 
the time to which the annotation corresponds) are extremely 
useful to, for example, enable someone not familiar with a 
videotape to identify the content of the videotape, and, 
further, the location of particular content within the 
35 videotape. The time scale illustrated in FIG. 9 displays the 
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number of - moving from left to right in FIG, 9 - hours, 
minutes, seconds and frames (within a second) that have 
passed since the beginning of a use of the visual user 
interface 900 (which, in a typical use of the visual user 
5 interface 900, will correspond to the time that has passed 
since the beginning of a videotape) . Other time scales can 

be used. Though useful, a time scale need *not necessarily be 

present in the annotation space 904; the user could also note 
the time of an annotation as part of the annotation. 

10 The annotation space 904 can be implemented using any 

appropriate apparatus and methods. For example, if the 
visual user interface 900 is displayed on a touchscreen, a 
stylus or the user's finger can be used to write notes in the 
annotation space 904. Such an embodiment of the invention is 

15 illustrated in FIG. 9, which shows a stylus 905 that can be 
used both to make notes in the annotation space 904 and to 
change the control scene 902. The touchscreen enables 
ascertainment of the patterns written in the annotation 
space 904. Software embodying handwriting recognition 

20 methods can then be used to interpret the patterns as words, 
numbers and symbols. Touchscreens and handwriting 
recognition software that have previously been developed and 
used in computational devices such as, for example, personal 
digital assistants (PDAs) can be used to implement an 

25 annotation space in a visual user interface according to the 
invention . 

The annotation space 904 can also be implemented by 
enabling the user to use a conventional keyboard to type 
notes into the annotation space 904. Other ways of 

30 implementing the annotation space 904 are possible, as can be 
appreciated by those skilled in the art. However the 
annotation space 904 is implemented, it is, in general, 
preferable to enable interaction with the control scene 902 
and the annotation space 9 04 via the same input device, so 

3 5 that the user can easily switch back and forth between the 
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two . 



The presence of an annotation space as part of the same 
user interface that enables control of a video device enables 
a single person to easily both operate the video device and 
5 make notes regarding the content of a sequence of filmed 

scenes acquired or displayed by the video device. A user of 
the interface can quickly move back and forth between 
controlling the video device via the control scene 902 and 
making notes in the annotation space 904, thus enabling 

10 annotations to be made contemporaneously with control of the 
video device. Moreover, when the device control portion of 
the interface is implemented in accordance with the aspect of 
Che invention described above with respect to FIGS. 1 
through 8B, the ease with which that portion of the interface 

15 enables device control contributes to the ease with which the 
complete interface can be used for contemporaneous device 
control and annotation, because the user is left with more 
time and attention to devote to making annotations. In 
contrast, previous ways of annotating a sequence of filmed 

20 scenes as they are acquired either necessitated two people 
(one to operate the video camera and the other to take 
notes) , or unduly interfered with the operation of the video 
camera. 



25 handwriting recognition software in a user interface for 

enabling annotations of a sequence of filmed scenes provides 
wide latitude in expression, thus avoiding a limitation of 
previous systems that enable selection of an annotation from 
a predetermined set of *'canned" annotations. Such a user 

3 0 interface also avoids the problems associated with voice 
annotation systems, e.g., disruption of the event or scene 
being filmed, and difficulty in searching through the 
annotations, (However, a visual user interface according to 
the invention can, if desired, be implemented with a voice 

35 annotation system. ) 



Additionally, the use of a touchscreen, stylus and 
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A visual user interface according to the invention can 
also be used to play back a sequence of filmed scenes that 
has previously been acquired using the visual user interface. 
Recorded camera metric information that correlates positions 
5 of the movable camera mount with filming directions, 

orientations and/or positions of the video camera during 
acquisition of filmed scenes can be used during "playback to" 
locate and scale the filmed scene properly with respect to a 
displayed target scene, so that, for example, the sequence of 

10 filmed scenes can be viewed in the context of a target scene 
that represents a panoramic scene within which the filmed 
scenes were acquired. As can be appreciated, use of the 
visual user interface according to the invention during 
playback provides a much enhanced viewing experience as 

15 compared to that obtained using conventional camcorders or 
television monitors . 

Additionally, a visual user interface according to the 
invention can be used for editing or selective playback of 
the previously acquired filmed scenes. In general, the 

20 content of the control scene can be controlled in any manner 
as described above to identify particular content from the 
sequence of filmed scenes to either display or store. For 
example, a stylus and touchscreen can be used as described 
above with respect to FIGS. 7A through 7E to position the 

25 control scene at one or more locations, an appropriate user 
interface mechanism (e.g., any of a variety of appropriate 
graphical user interface mechanisms, such as GUI pushbuttons 
or pulldown menus) being used to specify that filmed scenes 
having content that is partly or entirely subsumed within the 

30 area defined by the specified control scenes be stored or 
displayed. The stylus and touchscreen can also be used to 
specify, rather than locations, particular types of movements 
(e.g., pans, tilts, zooms) of the control scene, the user 
interface mechanism being used to indicate that sequences of 

3 5 filmed scenes corresponding to such movements be stored or 
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displayed. The control scene can also be controlled to _ 
specify any desired combination of control scene locations 
and/or movements to specify filmed scenes to be displayed or 
stored . 

5 A visual user interface according to the invention can 

also be integrated with a video device that can be 
automatically controlled without user intervention (e.g. a" 
video camera, such as the Sony EVI-D30 video camera). An 
appropriate user interface can be used to enable a user to, 

10 if desired, specify that the video device operate in 

accordance with a predetermined method for automatically 
controlling the video device (e.g., motion- tracking or color- 
tracking methods). For example, when a visual user interface 
according to the invention is used to control the operation 

15 of a video camera, such integration of automatic video camera 
control with the visual user interface can be useful to 
enable a camera operator to specify automatic operation of 
the camera during a period in which the user desires to make 
extensive annotations regarding the scenes being filmed. 

20 Various embodiments of the invention have been 

described. The descriptions are intended to be illustrative, 
not limitative. Thus, it will be apparent to one skilled in 
the art that certain modifications may be made to the 
invention as described above without departing from the scope 

25 of the claims set out below. 
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I claim: 

1. A visual user interface for use in controlling the 
interaction of a device with a spatial region, comprising: 

means for displaying a target space and a control 
5 space, wherein an overlapping portion of the control 

space is within the target space, the content of the 
' overlapping po'rtion of the contrrol space" being ^ 
substantially the same as the content of the target 
space that would otherwise be present at the location of 
10 the overlapping portion of the control space, such that 

the content of the target space provides context for at 
least the overlapping portion of the control space; and 

means for enabling a user to change the state of 
the display of the control space, wherein the device is 
15 controlled in response to the state of the display of 

the control space. 

2. A visual user interface as in Claim 1, wherein the 
target space and control space are a target scene and control 
scene , respectively . 

20 3. A visual user interface as in Claim 2, wherein the 

device is a video device that is adapted to film and/or to 
display a sequence of filmed scenes, the filmed scenes being 
controlled in response to the state of the display of the 
control scene. 

25 4. A visual user interface as in Claim 3, wherein the 

content of the control scene is substantially the same as the 
content of a current filmed scene. 

5. A visual user interface as in Claim 3, further 
comprising means for displaying a current filmed scene. 
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6. A visual user interface as in Claim 5, wherein^the 
means for displaying a target scene and a control scene, and 
the means for displaying a current filmed scene are embodied 
in the same device. 

5 7. A visual user interface as in Claim 6, wherein the 

display of the filmed scene does not overlap either the - 
display of the target scene or the display of the control 
scene . 

8. A visual user interface as in Claim 5, wherein the 
10 display of the current filmed scene has an unchanging size. 

9. A visual user interface as in Claim 2, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the size of the control scene with respect to the target 

15 scene. 

10. A visual user interface as in Claim 2, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the shape of the control scene with respect to the target 

20 scene. 

11. A visual user interface as in Claim 2, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the angular orientation of the control scene with respect to 

25 the target scene , 
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12. A visual user interface as in Claim 2, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the location of the control scene with respect to the target 
5 scene. 



13. A visual user interface as in Claim 2, wherein: 
the means for displaying a target scene and a 

control scene further comprises a touchscreen; and 

the means for enabling a user to change the state 
10 of the display of the control scene further comprises 

the touchscreen and means for contacting the 

touchscreen. 



14. A visual user interface as in Claim 2, wherein the 
means for enabling a user to change the state of the display 
15 of the control scene is adapted to enable the user to change 
visual parameters of the control scene. 



15. A visual user interface as in Claim 14, wherein the 
visual parameters include focus, tint, contrast and/or 
brightness , 



20 16. A visual user interface as in Claim 2, wherein the 

overlapping portion of the control scene can comprise less 
than the entire control scene. 



17, A visual user interface as in Claim 2, wherein the 
overlapping portion of the control scene comprises the entire 
25 control scene. 



18. A visual user interface as in Claim 2, wherein the 
area of the target scene is greater than the area of the 
control scene. 
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19. A visual user interface as in Claim 2, wherein jche 
display of the control scene is a dynamic display. 

20, A visual user interface as in Claim 2, wherein the 
display of the control scene is a static display. 

5 21. A visual user" inter face as' in' Claim 2, wherein the~ 

display of the target scene is a static display. 

22. A visual user interface as in Claim 2, wherein the 
display of the target scene is a dynamic display. 

23. A visual user interface as m Claim 2, wherein the 
10 content: of the target scene can change in response to changes 

in the location of the control scene so as to maintain an 
overlapping portion of the control scene within the target 
scene . 

24. A visual user interface as in Claim 2, wherein the 
15 interaction of multiple devices with the spatial region can 

be controlled using the visual user interface. 

25. A visual user interface as in Claim 1, wherein: 
the device is a light source; and 

the interaction of the light source with the 
20 spatial region comprises the intensity of light emitted 

from the light source at each point within the spatial 
region . 

26. A visual user interface for use in controlling the 
interaction of a device with a spatial region, comprising: 

25 means for displaying a target spacer- 

means for displaying a control spacer- 
means for relating the state of the control space 
to the state of the target space; 
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means for enabling a user to change the state of_ 
the display of the control space; and 



5 



means for controlling the device in response to the 
relationship between the state of the control space and 
the state of the target space 



"27, A visual user interface as in Claim 26. wherein the 
target space and control space are a target scene and control 
scene , respectively . 

28. A visual user interface as in Claim 27, wherein the 
10 means for displaying a target scene and the means for 

displaying a control scene are embodied in the same device. 

29 . A visual user interface as in Claim 27 , wherein the 
device is a video device that is adapted to film and/or to 
display a sequence of filmed scenes, the filmed scenes being 

15 controlled in response to the state of the display of the 
control scene. 

30. A visual user interface as in Claim 29, wherein the 
content of the control scene is substantially the same as the 
content of the filmed scene. 

20 31. A visual user interface as in Claim 28, further 

comprising means for displaying a current filmed scene. 

32. A visual user interface as in Claim 31, wherein the 
means for displaying a target scene, the means for displaying 
a control scene and the means for displaying a current filmed 
25 scene are embodied in the same device. 
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33. A visual user interface as in Claim 27, wherein the 

means for enabling a user to change the state of the display 

of the control scene is adapted to enable the user to change 

the size of the control scene with respect to the target 



34. A visual user interface as in Claim 27, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the shape of the control scene with respect to the target 

10 scene, 

35. A visual user interface as in Claim 27, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the angular orientation of the control scene with respect to 

15 the target scene. 

36. A visual user interface as in Claim 27, wherein the 
means for enabling a user to change the state of the display 
of the control scene is adapted to enable the user to change 
the location of the control scene with respect to the target 

20 scene. 



5 scene. 



37. A visual user interface as in Claim 27, wherein: 



the means for displaying a control scene further 
comprises a touchscreen; and 



25 



the means for enabling a user to change the state 
of the display of the control scene further comprises 
the touchscreen and means for contacting the 



touchscreen . 



30 



38. A visual user interface, comprising: 

means for controlling the interaction of a device 
with a spatial region; and 
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means for enabling a user to record annotations _ 
regarding the interaction of the device with the spatial 
region, wherein the means for controlling the device and 
the means for enabling recordation of annotations are 
5 embodied in the same device or devices. 

39. A visual user interface as in Claim 38, wherein the 
device is a video device that is adapted to film and/or to 
display a sequence of filmed scenes. 

40. A visual user interface as in Claim 39, wherein the 
10 means for controlling the video device further comprises: 

means for displaying a target scene and a control 
scene, wherein an overlapping portion of the control 
scene is within the target scene, the content of the 
overlapping portion of the control scene being 
substantially the same as the content of the target 
scene that would otherwise be present at the location of 
the overlapping portion of the control scene, such that 
the content of the target scene provides context for at 
least the overlapping portion of the control scene; and 

means for enabling a user to change the state of 
the display of the control scene, wherein the video 
device is controlled in response to the state of the 
display of the control scene. 

41. A visual user interface as in Claim 40, wherein: 
the means for displaying a target scene and a 

control scene further comprises a touchscreen; 

the means for enabling a user to change the state 
of the display of the control scene further comprises 
the touchscreen and means for contacting the 
touchscreen; and 

the annotation means further comprises the 
touchscreen and means for contacting the touchscreen. 



20 



25 
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42. A visual user interface as in Claim 38, whereixi the 
annotation means comprises means for enabling the user to 
handwrite an annotation . 

43. A visual user interface as in Claim 38, wherein the 
5 annotation means further comprises means for displaying a 

time scale that can be related to a duration of time of 
device interaction with the spatial region. 

44. A method for use in controlling the interaction of 
a device with a spatial region, comprising: 

10 displaying a target space and a control space, 

wherein an overlapping portion of the control space is 
within the target space, the content of the overlapping 
portion of the control space being substantially the 
same as the content of the target space that would 

15 otherwise be present at the location of the overlapping 

portion of the control space, such that the content of 
the target space provides context for at least the 
overlapping portion of the control space; and 

enabling a user to change the state of the display 

20 of the control space, wherein the device is controlled 

in response to the state of the display of the control 
space . 

45. A method as in Claim 44, wherein the target space 
and control space are a target scene and control scene, 

25 respectively. 

46. A method as in Claim 45, wherein the device is a 
video device that is adapted to film and/or to display a 
sequence of filmed scenes, the filmed scenes being controlled 
in response to the display of the state of the control scene. 
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47. A method as in Claim 46, further comprising 
displaying a current filmed scene. 

48. A method as in Claim 45, wherein the step of 
enabling a user to change the state of the display of the 

5 control scene further comprises the step of enabling the user 
to change the size of the control scene with' respect to the 
target scene. 

49. A method as in Claim 45, wherein the step of 
enabling a user to change the state of the display of the 

10 control scene further comprises the step of enabling the user 
to change the shape of 'the control scene with respect to the 
target scene . 

50. A method as in Claim 45, wherein the step of 
enabling a user to change the state of the display of the 

15 control scene further comprises the step of enabling the user 
to change the angular orientation of the control scene with 
respect to the target scene. 

51. A method as in Claim 45, wherein the step of 
enabling a user to change the.,,state of the display of the 

20 control scene further comprises the step of enabling the user 
to change the location of the control scene with respect to 
the target scene, 

52. A method as in Claim 45, wherein the step of 
enabling a user to change the state of the display of the 

25 control scene further comprises the step of enabling the user 
to change visual parameters of the control scene. 
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53. A method as in Claim 52, wherein the step of 
enabling a user to change visual parameters further comprises 
the step of enabling the user to change the focus, tint, 
contrast and/or brightness of the display of the control 



54. A method as in Claim 45, further comprising the 
step of changing the content of the target scene in response 
to changes in the location of the control scene so as to 
maintain an overlapping portion of the control scene within 

10 the target scene. 

55. A method as in Claim 45, wherein the target scene 
is a composite scene that is constructed from multiple filmed 
scenes, the method further comprising the step of acquiring 
one or more of the filmed scenes used to construct the target 

15 scene during the steps of displaying and enabling, 

56. A method as in Claim 45. further comprising the 
step of modifying in a predetermined manner the control of 
the device in response to the state of the display of the 
control scene. 



5 scene. 
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